Adaptive Metric Kernel

Kernel smoothing is a widely used non-parametric pattern recognition technique. By nature, it suuers from the curse of dimensionality and is usually diicult to apply to high input dimensions. In this contribution, we propose an algorithm that adapts the input metric used in multivariate regression by minimising a cross-validation estimate of the generalisation error. This allows to automatically adjust the importance of diierent dimensions. The improvement in terms of modelling performance is illustrated on a variable selection task where the adaptive metric kernel clearly outperforms the standard approach. OVERVIEW Neural Networks are often referred to as a non-parametric model. Their popularity is probably in part linked to the fact that there are many cases in which one tries to model some input-output relationship on the basis of empirical data, without any parametric model of the underlying phenomenon. Kernel methods are the archetypal non-parametric method 7], a well-known tool for eg pattern recognition 10]. They have been \re-invented" in the neural networks literature on several occasions 11, 12], for density estimation , classiication and regression purposes. However, it has been consistently noted that they suuer badly from the \curse-of-dimensionality", ie produce poor estimators when the input dimension increases. In this contribution, we will address a possible improvement on the traditional multivariate kernel method. We will be mainly concerned with regression estimation, but the method presented below applies to classiication tasks in a straightforward manner. We will rst present some general result on the uni-and multivariate kernel regression estimation. We then introduce our method, based on the adaptive estimation of the feature space metric used by the kernels. We perform a number of experiments on a regression task where a number of irrelevant dimensions are added to the input space. The superiority of the adaptive metric scheme is illustrated by the fact that its